Skip to content

fix: make usage chunk in stream mode of gemini compatible with openai #1503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

fix: make usage chunk in stream mode of gemini compatible with openai #1503

wants to merge 10 commits into from

Conversation

@hustxiayang
Copy link
Contributor

@hustxiayang hustxiayang commented Nov 6, 2025 •
edited by yuzisun
Loading

Description

Users found the simply use "usage" information does not work for streaming responses of gemini models.
image

This is because for openai models, the usage chunk would be a separate chunk. For example, this is an example response from gpt-4o:

...

chunk=ChatCompletionChunk(id='chatcmpl-CYv6DPkWfT1xrsS2ySOoRztQKnZDg', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None, content_filter_result={'error': {'code': 'content_filter_error', 'message': 'The contents are not filtered'}})], created=1762438677, model='azure.gpt-4o', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_4a331a0222', usage=None, obfuscation='2xP')


chunk=ChatCompletionChunk(id='chatcmpl-CYv6DPkWfT1xrsS2ySOoRztQKnZDg', choices=[], created=1762438677, model='azure.gpt-4o', object='chat.completion.chunk', service_tier=None, system_fingerprint='fp_4a331a0222', usage=CompletionUsage(completion_tokens=17, prompt_tokens=12, total_tokens=29, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)), obfuscation='xY40HsJr')

There is a finish_reason chunk, and then a usage chunk.

Thus, want to make it compatible with Openai. (Actually, in anthropic translation, it's already compatible)

@hustxiayang hustxiayang requested a review from a team as a code owner November 6, 2025 14:21
@dosubot dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Nov 6, 2025
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Nov 6, 2025
@hustxiayang hustxiayang changed the title fix: make usage chunk similar to Openai. fix: make usage chunk in stream mode of gemini compatible with openai Nov 6, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Nov 6, 2025
@codecov-commenter
Copy link

codecov-commenter commented Nov 6, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 61.90476% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.24%. Comparing base (a2ba2c9) to head (8440419).

Files with missing lines Patch % Lines
internal/translator/openai_gcpvertexai.go 61.90% 4 Missing and 4 partials ⚠️

❌ Your patch status has failed because the patch coverage (61.90%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1503      +/-   ##
==========================================
- Coverage   83.26%   83.24%   -0.02%     
==========================================
  Files         137      137              
  Lines       12059    12069      +10     
==========================================
+ Hits        10041    10047       +6     
- Misses       1411     1413       +2     
- Partials      607      609       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hustxiayang
Copy link
Contributor Author

hustxiayang commented Nov 6, 2025

/retest

@nacx
Copy link
Contributor

nacx commented Nov 11, 2025

@yuzisun can you take a look at this one?

hustxiayang and others added 6 commits November 29, 2025 18:06
@hustxiayang
fix-usage-chunk
fef4558 
Signed-off-by: yxia216 <[email protected]>
@hustxiayang @yuzisun
fix: finish reason should be tool calls when the model responded with…
ab0b9ea 
… a tool call (envoyproxy#1486)

**Description**

Finish reason should be tool calls if the model returns a tool call
response. In vertex api, there is no tool call finish reason, thus need
a work around to make it compatible.

---------

Signed-off-by: yxia216 <[email protected]>
Co-authored-by: Dan Sun <[email protected]>
Signed-off-by: yxia216 <[email protected]>
@mathetake @hustxiayang
refactor: decouples backendauth & headermutator from extproc (envoypr…
3cb73bc 
…oxy#1491)

**Description**

This decouples backendauth & headermutator packages from extproc
specifics. As we are looking to migrate to dynamic modules, this is a
necessary refactoring work to make the code as reusable as possible.

**Related Issues/PRs (if applicable)**

Preliminary for envoyproxy#90

---------

Signed-off-by: Takeshi Yoneda <[email protected]>
Signed-off-by: yxia216 <[email protected]>
@hustxiayang
fix-tests
7f97a77 
Signed-off-by: yxia216 <[email protected]>
@hustxiayang
rebase
98cf408 
Signed-off-by: yxia216 <[email protected]>
@hustxiayang
rebase
30a3391 
Signed-off-by: yxia216 <[email protected]>
@hustxiayang
Copy link
Contributor Author

hustxiayang commented Dec 1, 2025

/retest

1 similar comment
@hustxiayang
Copy link
Contributor Author

hustxiayang commented Dec 1, 2025

/retest

@hustxiayang
Copy link
Contributor Author

hustxiayang commented Dec 5, 2025

/retest

1 similar comment
@hustxiayang
Copy link
Contributor Author

hustxiayang commented Dec 5, 2025

/retest

yuzisun
yuzisun reviewed Dec 5, 2025
View reviewed changes
internal/translator/openai_gcpvertexai.go Outdated
}

if span != nil {
span.RecordResponseChunk(openAIChunk)
Copy link
Contributor

@yuzisun yuzisun Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like copy/paste error, should be usageChunk ?

Copy link
Contributor Author

@hustxiayang hustxiayang Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated! Thanks a lot for the comment!

@hustxiayang
fix-comments
8440419 
Signed-off-by: yxia216 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuzisun yuzisun yuzisun left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hustxiayang @codecov-commenter @nacx @yuzisun @mathetake